class: center, middle, inverse, title-slide .title[ # Differences-in-Differences ] .subtitle[ ## The Most Commonly Applied Research Design in Modern Econometrics ] --- # Difference-in-Differences - Today we will talk about difference-in-differences (DID), which is a way of using within variation in a more deliberate way in order to identify the effect we want - All we need is a treatment that *goes into effect* at a particular time, and we need a group that is *treated* and a group that is *not* -- - Then, we compare the within-variation for the treated group vs. the within-variation for the untreated group -- - This is the effect ! --- # Difference-in-Differences - Because the requirements to use it are so low, DID is used a *lot* -- - Any time a policy is enacted but isn't enacted everywhere at once? DID! --- # Difference-in-Differences - The question DID tries to answer is **"what was the effect of (some policy) on the people who were affected by it?"** -- - We have some data on the people who were affected both before the policy went into effect and after -- - Can we just compare the mean before and after? -- - *No, because things change over Time for reasons unrelated to Treatment* <!-- --> --- # Difference-in-Differences - Why not just control for `Time`? -- - But we can't! It would "wash out" all the variation! -- - You're either Before and Untreated, or After and Treated. - So if you control for Time, you're comparing people with the same values of Time - who must also have the same values of Treatment! - So you can't compare Treated and Untreated to get the effect! --- # Difference-in-Differences - The solution is to bring in a *control group* who is `Untreated` in both `Before` and `After` periods -- - There's more to control for (we have to control for Group differences too - isolate within variation), but this ALLOWS us to control for `Time` <!-- --> --- # Difference-in-Difference - We control for Group by isolating within variation (comparing After to Before) -- - Then we control for Time by comparing those two sources of within variation -- - We ask *how much more increase* was there in `Treatment` than `Control` group? -- - The Control change is probably the increase you could have expected regardless of Treatment -- - So anything on *top of that* is the effect of `Treatment` - Now we've controlled for Group and Time, identifying the effect! --- # Example - Let's say we have a pill that's supposed to make you taller - Give it to a kid Adelaide who is 48 inches tall - Next year they're 54 inches tall - a six inch increase! But they probably would have grown some anyway without the pill. Surely the pill doesn't make you six inches taller. - SO we compare them to their twin Bella, who started at 47 inches but we DON'T give a pill to - Next year that twin is 51 inches tall - a four inch increase. So Adelaide probably would have grown about 4 inches without the pill. - So the pill boosted her by `\((54 - 48) - (51 - 47) = 6 - 4 = 2\)` additional inches - "Adelaide (who was `Treated`) grew by two *more inches* than Bella (who was `Untreated`) did over the same period of time, so the pill made Adelaide grow by two inches" - **That's DID!** --- # Example <!-- --> --- # Example <!-- --> --- # Difference-in-Differences What *changes* are included in each value? - Untreated Before: Untreated Group Mean - Untreated After: Untreated Group Mean + Time Effect - Treated Before: Treated Group Mean - Treated After: Treated Group Mean + Time Effect + Treatment Effect - Untreated After - Before = Time Effect - Treated After - Before = Time Effect + Treatment Effect - DID = (Treated After - Before) - (Untreated After - Before) = Treatment EFfect --- # Concept Checks - Why do we need a control group? What does this let us do? - What do we need to assume is true about our control group? - In 2015, a new, higher minimum wage went into effect in Seattle, but this increase did not occur in some of the areas surrounding Seattle. How might you use DID to estimate the effect of this minimum wage change on employment levels? --- # Difference-in-Difference - Of course, this only uses four data points - Usually these four points would be four means from lots of observations, not just two people in two time periods - How can we do this and get things like standard errors, and perhaps include controls? + Use OLS of course! --- # Difference-in-Differences - We can use what we know about binary variables and interaction terms to get our DID `$$Y_{it} = \beta_0 + \beta_1After_t + \beta_2Treated_i + \beta_3After_t*Treated_i + \varepsilon_{it}$$` where `\(After_t\)` is a binary variable for being in the post-treatment period, and `\(Treated_t\)` is a binary variable for being in the treated group ``` ## Person Time Height After Treated ## 1 Adelaide Before 48 FALSE TRUE ## 2 Adelaide After 54 TRUE TRUE ## 3 Bella Before 47 FALSE FALSE ## 4 Bella After 51 TRUE FALSE ``` --- # Difference-in-Differences - How can we interpret this using what we know? `$$Y_{it} = \beta_0 + \beta_1After_t + \beta_2Treated_i + \beta_3After_tTreated_i + \varepsilon_{it}$$` -- - `\(\beta_0\)` is the prediction when `\(Treated_i = 0\)` and `\(After_t = 0\)` `\(\rightarrow\)` the mean for Untreated group Before the treatment was implemented! - `\(\beta_1\)` is the *difference between* Before and After for `\(Treated_i = 0\)` `\(\rightarrow\)` Untreated (After - Before) - `\(\beta_2\)` is the *difference between* Treated and Untreated for `\(After_t = 0\)` `\(\rightarrow\)` Before (Treated - Untreated) - `\(\beta_3\)` is *how much bigger the Before-After difference* is for `\(Treated_i = 1\)` than for `\(Treated_i = 0\)` `\(\rightarrow\)` (Treated: After - Before) - (Untreated: After - Before) = DID! --- # Graphically <!-- --> --- # Design vs. Regression - There is a distinction between *regression model* and *research design* - We have a model with an interaction term - Not all models with interaction terms are DID! -- - It's DID because it's an interaction between treated/control and before/after - If you don't have a before/after, or you don't have a control group, that same setup may tell you something interesting but it won't be DID! --- # Example - The Earned Income Tax Credit was increased in 1993. This may increase chances single mothers (treated) return to work, but likely not affect single non-moms (control) - Does this program help moms get back to work? ```r df <- read.csv('http://nickchk.com/eitc.csv') %>% mutate(after = year >= 1994, treated = children > 0) df %>% group_by(after, treated) %>% summarize(proportion_working = mean(work)) ``` ``` ## # A tibble: 4 × 3 ## # Groups: after [2] ## after treated proportion_working ## <lgl> <lgl> <dbl> ## 1 FALSE FALSE 0.575 ## 2 FALSE TRUE 0.446 ## 3 TRUE FALSE 0.573 ## 4 TRUE TRUE 0.491 ``` --- # Example - We can do it by just comparing the points, like we did with Adelaide and Bella - This will give us the DID estimate: The EITC increase increases the probability of working by 4.7 percentage points - But not standard errors, or the ability to include controls easily ```r means <- df %>% group_by(after, treated) %>% summarize(proportion_working = mean(work)) %>% pull(proportion_working) (means[4] - means[2]) - (means[3] - means[1]) ``` ``` ## [1] 0.04687313 ``` --- # Let's try OLS! ```r library(modelsummary) did_reg <- feols(work ~ after*treated, data = df) # DiD regression # interaction term after*treated includes the main effects too (after and treated) msummary(did_reg,stars = TRUE) ``` <table style="NAborder-bottom: 0; width: auto !important; margin-left: auto; margin-right: auto;" class="table"> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:center;"> Model 1 </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> (Intercept) </td> <td style="text-align:center;"> 0.575*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (0.009) </td> </tr> <tr> <td style="text-align:left;"> afterTRUE </td> <td style="text-align:center;"> −0.002 </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (0.013) </td> </tr> <tr> <td style="text-align:left;"> treatedTRUE </td> <td style="text-align:center;"> −0.129*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (0.012) </td> </tr> <tr> <td style="text-align:left;"> afterTRUE × treatedTRUE </td> <td style="text-align:center;"> 0.047** </td> </tr> <tr> <td style="text-align:left;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> (0.017) </td> </tr> <tr> <td style="text-align:left;"> Num.Obs. </td> <td style="text-align:center;"> 13746 </td> </tr> <tr> <td style="text-align:left;"> R2 </td> <td style="text-align:center;"> 0.013 </td> </tr> <tr> <td style="text-align:left;"> R2 Adj. </td> <td style="text-align:center;"> 0.012 </td> </tr> <tr> <td style="text-align:left;"> R2 Within </td> <td style="text-align:center;"> </td> </tr> <tr> <td style="text-align:left;"> R2 Pseudo </td> <td style="text-align:center;"> </td> </tr> <tr> <td style="text-align:left;"> AIC </td> <td style="text-align:center;"> 19777.8 </td> </tr> <tr> <td style="text-align:left;"> BIC </td> <td style="text-align:center;"> 19807.9 </td> </tr> <tr> <td style="text-align:left;"> Log.Lik. </td> <td style="text-align:center;"> −9884.917 </td> </tr> <tr> <td style="text-align:left;"> Std.Errors </td> <td style="text-align:center;"> IID </td> </tr> </tbody> <tfoot><tr><td style="padding: 0; " colspan="100%">  + p &lt; 0.1, * p &lt; 0.05, ** p &lt; 0.01, *** p &lt; 0.001</td></tr></tfoot> </table> --- # Concept Checks - There are four coefficients on the previous slide. Interpret them carefully in a sentence each. --- # Does it Work? - We have a causal effect as long as *the only reason the gap changed* was the treatment -- - If Adelaide would have grown six inches *anyway*, then the gap would have grown by 2, pill or not. The pill did nothing in that case! But we don't know that and mistakenly say it helped her grow by 2 -- - In DID, we need to **assume** that there's no uncontrolled endogenous variation *across this particular before/after time change* (**Parralel trends assumption**) --- # Parallel Trends Assumption - "Parallel trends" assumption mean that nothing else changes at the same time -- - Again, this assumes that, *if the Treatment hadn't happened to anyone*, the gap between the two would have stayed the same -- - Sometimes people check whether this assumption is plausible by seeing if *prior trends* are the same for `Treated` and `Untreated` - if we have multiple pre-treatment periods, was the gap changing a lot during that period? --- # Prior Trends - Let's see how that EITC example looks in the lead up to 1994 - They look like the gap between them is pretty constant before 1994! They move up and down but the *gap* stays the same. That's good. <!-- --> --- # Prior Trends - Formally, prior trends being the same tells us nothing about parallel trends - But it can be suggestive. Going back to the height pill example, what if instead of comparing Adelaide and Bella, child twins, we compared Adelaide to *me*? - Seeing the gap closing *anyway* in previous years would be a pretty good clue that it's not just the pill <!-- --> --- # Parallel Trends - **Just because *prior trends* are equal doesn't mean that *parallel trends* holds.** -- - *Parallel trends assumption* is about what the before-after change *would have been* - we can't see that! -- - For example, let's say we want to see the effect of online teaching on student test scores, using COVID school shutdowns to get a Before/After - As of March/April 2020, some schools had gone online (Treated) and others hadn't (Untreated) -- - Test score trends were probably pretty similar in the Before periods (Jan/Feb 2020), so prior trends are likely the same -- - But LOTS of stuff changed between Jan/Feb and Mar/Apr, like, uh, Coronavirus, lockdowns, etc. not just online teaching! SO *parallel trends* likely wouldn't hold --- # Concept Checks - Go back to the Seattle minimum wage effect example from the first Concept Check slide. Clearly state what the parallel trends assumption means in this context. --- # Swirl - Let's swirl - Do the Difference-in-differences swirl --- ### Kessler and Roth (2014) “Don’t Take ’No’ for an Answer: An Experiment with Actual Organ Donor Registrations.” ```r #install.packages("causaldata") library(tidyverse); library(modelsummary); library(fixest) od <- causaldata::organ_donations # Treatment variable od<-od%>% mutate(Treated= ifelse( State == 'California',1,0 ), After=ifelse(Quarter %in% c('Q32011','Q42011','Q12012'),1,0)) # feols clusters by the first # fixed effect by default, no adjustment necessary clfe_new <- feols(Rate ~ Treated*After, data = od) msummary(clfe_new, cluster = ~State, stars = c('*' = .1, '**' = .05, '***' = .01)) ``` --- <table style="NAborder-bottom: 0; width: auto !important; margin-left: auto; margin-right: auto;" class="table"> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:center;"> Model 1 </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> (Intercept) </td> <td style="text-align:center;"> 0.445*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (0.031) </td> </tr> <tr> <td style="text-align:left;"> Treated </td> <td style="text-align:center;"> −0.174*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (0.031) </td> </tr> <tr> <td style="text-align:left;"> After </td> <td style="text-align:center;"> 0.014** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (0.006) </td> </tr> <tr> <td style="text-align:left;"> Treated × After </td> <td style="text-align:center;"> −0.022*** </td> </tr> <tr> <td style="text-align:left;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> (0.006) </td> </tr> <tr> <td style="text-align:left;"> Num.Obs. </td> <td style="text-align:center;"> 162 </td> </tr> <tr> <td style="text-align:left;"> R2 </td> <td style="text-align:center;"> 0.054 </td> </tr> <tr> <td style="text-align:left;"> R2 Adj. </td> <td style="text-align:center;"> 0.036 </td> </tr> <tr> <td style="text-align:left;"> R2 Within </td> <td style="text-align:center;"> </td> </tr> <tr> <td style="text-align:left;"> R2 Pseudo </td> <td style="text-align:center;"> </td> </tr> <tr> <td style="text-align:left;"> AIC </td> <td style="text-align:center;"> −149.8 </td> </tr> <tr> <td style="text-align:left;"> BIC </td> <td style="text-align:center;"> −137.4 </td> </tr> <tr> <td style="text-align:left;"> Log.Lik. </td> <td style="text-align:center;"> 78.895 </td> </tr> <tr> <td style="text-align:left;"> Std.Errors </td> <td style="text-align:center;"> IID </td> </tr> </tbody> <tfoot><tr><td style="padding: 0; " colspan="100%">  * p &lt; 0.1, ** p &lt; 0.05, *** p &lt; 0.01</td></tr></tfoot> </table> --- ### Paralel trend assumption ```r library(tidyverse); library(modelsummary); library(fixest) od <- causaldata::organ_donations %>% # Use only pre-treatment data filter(Quarter_Num <= 3) # Create our fake treatment variables od <- od %>% mutate(FakeTreat1 = State == 'California' & Quarter %in% c('Q12011','Q22011'), FakeTreat2 = State == 'California' & Quarter == 'Q22011') # Run the same model we did before but with our fake treatment clfe1 <- feols(Rate ~ FakeTreat1 | State + Quarter, data = od) clfe2 <- feols(Rate ~ FakeTreat2 | State + Quarter, data = od) msummary(list(clfe1,clfe2), stars = c('*' = .1, '**' = .05, '***' = .01)) ``` --- ### Paralel trend assumption <table style="NAborder-bottom: 0; width: auto !important; margin-left: auto; margin-right: auto;" class="table"> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:center;"> Model 1 </th> <th style="text-align:center;"> Model 2 </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> FakeTreat1TRUE </td> <td style="text-align:center;"> 0.006 </td> <td style="text-align:center;"> </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (0.005) </td> <td style="text-align:center;"> </td> </tr> <tr> <td style="text-align:left;"> FakeTreat2TRUE </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> −0.002 </td> </tr> <tr> <td style="text-align:left;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> (0.003) </td> </tr> <tr> <td style="text-align:left;"> Num.Obs. </td> <td style="text-align:center;"> 81 </td> <td style="text-align:center;"> 81 </td> </tr> <tr> <td style="text-align:left;"> R2 </td> <td style="text-align:center;"> 0.994 </td> <td style="text-align:center;"> 0.994 </td> </tr> <tr> <td style="text-align:left;"> R2 Adj. </td> <td style="text-align:center;"> 0.990 </td> <td style="text-align:center;"> 0.990 </td> </tr> <tr> <td style="text-align:left;"> R2 Within </td> <td style="text-align:center;"> 0.002 </td> <td style="text-align:center;"> 0.000 </td> </tr> <tr> <td style="text-align:left;"> R2 Pseudo </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> </tr> <tr> <td style="text-align:left;"> AIC </td> <td style="text-align:center;"> −421.7 </td> <td style="text-align:center;"> −421.5 </td> </tr> <tr> <td style="text-align:left;"> BIC </td> <td style="text-align:center;"> −349.8 </td> <td style="text-align:center;"> −349.7 </td> </tr> <tr> <td style="text-align:left;"> Log.Lik. </td> <td style="text-align:center;"> 240.838 </td> <td style="text-align:center;"> 240.766 </td> </tr> <tr> <td style="text-align:left;"> Std.Errors </td> <td style="text-align:center;"> by: State </td> <td style="text-align:center;"> by: State </td> </tr> <tr> <td style="text-align:left;"> FE: State </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> X </td> </tr> <tr> <td style="text-align:left;"> FE: Quarter </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> X </td> </tr> </tbody> <tfoot><tr><td style="padding: 0; " colspan="100%">  * p &lt; 0.1, ** p &lt; 0.05, *** p &lt; 0.01</td></tr></tfoot> </table> --- ### Paralel trend assumption ```r library(tidyverse); library(fixest) od <- causaldata::organ_donations # Treatment variable od <- od %>% mutate(California = State == 'California') # Interact quarter with being in the treated group using # the fixest i() function, which also lets us specify # a reference period (using the numeric version of Quarter) clfe <- feols(Rate ~ i(Quarter_Num, California, ref = 3) | State + Quarter_Num, data = od) # And use coefplot() for a graph of effects coefplot(clfe) ``` --- ### Paralel trend assumption <!-- -->